1. Download and process NLDAS

Phil Savoy

5/17/2021

Source: vignettes/1 Download and process NLDAS.Rmd

Introduction

This article describes downloading and processing total incoming shortwave radiation (W m-2) using the StreamLightUtils package. Total incoming shortwave radiation (W m-2) is provided by the National Land Data Assimilation System (NLDAS) at hourly timesteps at 0.125 degree spatial resolution. There are two potential workflows available: 1.) download data for a single site, and 2.) bulk download of data for many sites.

1. Downloading and processing NLDAS data for a single site

Incoming shortwave radiation data at a single site can be downloaded using the function NLDAS_DL which has the following structure:

NLDAS_DL(save_dir, Site_ID, Lat, Lon, startDate)

  • save_dir The save directory for the downloaded data to be placed in. For example, “C:/”
  • Site_ID The site identifier (“Site_ID”)
  • Lat The site latitude
  • Lon The site longitude
  • startDate The start date for the downloaded data (“YYYY-MM-DD”)

Once the data has been downloaded it requires some processing to format date and time information and extract the relevant data. The downloaded NLDAS data can be processed using the function NLDAS_proc which has the following structure:

NLDAS_proc(read_dir, Site_IDs)

  • read_dir The directory containing the downloaded NLDAS data
  • Site_IDs The Site ID(s) (“Site_ID”) write_output A logical value indicating whether the output should be written to disk (write_output = TRUE) or returned to the R environment (write_output = FALSE). The default value is FALSE since for most datasets this is a suitable approach; however, for very large datasets (thousands of sites) it may be easier to write files to disk instead of storing them in the workspace.
  • save_dir An optional parameter to use only when write_output = TRUE that indicates the save directory for files to be placed in. For example, "C:/
#Set the download location (add your own directory)
  working_dir <- "C:/"
    
#Download NLDAS data at NC_NHC
  NLDAS_DL(
    save_dir = working_dir,
    Site_ID = "NC_NHC",
    Lat = 35.9925, 
    Lon = -79.0460, 
    startDate = "2017-01-01"
  )
  
#Process the downloaded data
  NLDAS_processed <- NLDAS_proc(
    read_dir = working_dir, 
    Site_IDs = "NC_NHC"
  )

2. Downloading and processing NLDAS data for multiple sites

Incoming shortwave radiation data at multiple sites can be downloaded using the function NLDAS_DL_bulk which has the following structure:

NLDAS_DL_bulk(save_dir, site_locs, startDate)

  • save_dir The save directory for the downloaded data to be placed in. For example, “C:/”
  • site_locs A table with Site_ID, Lat, and Lon, and startDate
  • startDate An optional parameter. By default, if nothing is provided the function assumes that site_locs has a column that contains startDate. Alternatively, a single startDate can be provided as an argument for the download (YYYY-MM-DD)

Recall from earlier that our table of site information has a column called “startDate”, so here the optional parameter is not used. If data fails to download for a site, the NLDAS_DL_bulk function will automatically check and retry downloading data for all sites with missing data. However, it is possible that data does not exist for a site. Consequently, it is prudent to confirm the successfully downloaded sites.

#Read in a table with initial site information
  sites <- data(NC_site_basic)

#Download NLDAS data at NC_NHC
  NLDAS_DL_bulk(
    save_dir = working_dir,
    site_locs = sites
  )

#List of successfully downloaded sites
  NLDAS_list <- stringr::str_sub(list.files(working_dir), 1, -11)
  
#Processing the downloaded NLDAS data
  NLDAS_processed <- StreamLightUtils::NLDAS_proc(read_dir = working_dir, NLDAS_list)